Systems and methods for detecting trailer angle

ABSTRACT

Systems and methods for detecting trailer angle are provided. In one aspect, an in-vehicle control system includes an optical sensor configured to be mounted on a tractor so as to face a trailer coupled to the tractor, the optical sensor further configured to generate optical data indicative of an angle formed between the trailer and the tractor. The system further includes a processor and a computer-readable memory in communication with the processor and having stored thereon computer-executable instructions to cause the processor to receive the optical data from the optical sensor, determine at least one candidate plane representative of a surface of the trailer visible in the optical data based on the optical data, and determine an angle between the trailer and the tractor based on the at least one candidate plane.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/181,020, filed Nov. 5, 2018, the disclosure of which is incorporatedherein by reference. Any and all applications for which a foreign ordomestic priority claim is identified in the Application Data Sheet asfiled with the present application are hereby incorporated by referenceunder 37 CFR 1.57.

BACKGROUND Technological Field

The described technology generally relates to systems and methods forautonomous driving, and more particularly, to detecting the anglebetween sections of an articulated vehicle.

Description of the Related Technology

In autonomous driving systems, the accurate perception and prediction ofthe surrounding driving environment and traffic participants are crucialfor making correct and safe decisions for control of the autonomous orhost vehicle. Additionally, the autonomous driving system must haveaccurate measurements of the current state of the vehicle being driven(also referred to as the ego vehicle), for example, of the speed,acceleration, road conditions, and location of the vehicle within thecurrent lane, etc. Certain vehicles, such as a semi-truck having atractor and at least one trailer, an articulated bus, a train, etc., mayhave a more complex state due to the additional degrees of freedomprovided by a pivot point(s) at which the trailer(s) are attached to thetractor or previous trailer. In this context, the accurate measurementof the angle between the sections of the articulate vehicle is asignificant factor used to define the state of the articulated vehicle.

SUMMARY OF CERTAIN INVENTIVE ASPECTS

One inventive aspect is an in-vehicle control system for a tractor andtrailer, comprising: an optical sensor configured to be mounted on thetractor and generate optical data indicative of an angle formed betweenthe trailer and the tractor; a processor; and a computer-readable memoryin communication with the processor and having stored thereoncomputer-executable instructions to cause the processor to: receive theoptical data, determine at least one candidate plane representative of asurface of the trailer, the candidate plane being visible in the opticaldata, and determine the angle between the trailer and the tractor basedat least on the at least one candidate plane.

Another inventive aspect is a non-transitory computer readable storagemedium having stored thereon instructions that, when executed, cause atleast one computing device to: receive optical data from an opticalsensor, the optical sensor configured to be mounted on a tractor andgenerate the optical data indicative of an angle formed between atrailer and the tractor; determine at least one candidate planerepresentative of a surface of the trailer, the candidate plane beingvisible in the optical data; and determine the angle between the trailerand the tractor based at least on the at least one candidate plane.

Yet another inventive aspect is a method for determining the distancebetween a vehicle and a lane, comprising: receiving optical data from anoptical sensor, the optical sensor configured to be mounted on a tractorand generate the optical data indicative of an angle formed between atrailer and the tractor; determining at least one candidate planerepresentative of a surface of the trailer, the candidate plane beingvisible in the optical data; and determining the angle between thetrailer and the tractor based at least on the at least one candidateplane.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example ecosystem including anin-vehicle control system and an image processing module in accordancewith aspects of this disclosure.

FIG. 2 is a simplified diagram of a semi-truck having a vehicleoperational subsystem configured to detect an angle between a tractorand a trailer of the semi-truck in accordance with aspects of thisdisclosure.

FIG. 3 illustrates an example block diagram of the vehicle operationalsubsystem for the autonomous control of the semi-truck that has theangle illustrated in FIG. 2 in accordance with aspects of thisdisclosure.

FIG. 4 illustrates an example method which can be used to determine theangle illustrated in FIG. 2 in accordance with aspects of thisdisclosure.

FIG. 5 illustrates another example method which can be used to determinetractor to trailer angle in accordance with aspects of this disclosure.

DETAILED DESCRIPTION OF CERTAIN INVENTIVE EMBODIMENTS Introduction toIn-Vehicle Control Systems

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the various embodiments. It will be evident, however,to one of ordinary skill in the art that the various embodiments may bepracticed without these specific details.

As described in various example embodiments, a system and method fordetecting trailer angle for an articulated vehicle are described herein.An example embodiment disclosed herein can be used in the context of anin-vehicle control system 150 in a vehicle ecosystem 101. In one exampleembodiment, an in-vehicle control system 150 with an image processingmodule 200 resident in a vehicle 105 can be configured like thearchitecture and ecosystem 101 illustrated in FIG. 1. However, it willbe apparent to those of ordinary skill in the art that the imageprocessing module 200 described herein can be implemented, configured,and used in a variety of other applications and systems as well.

With continuing reference to FIG. 1, a block diagram illustrates anexample ecosystem 101 in which an in-vehicle control system 150 and animage processing module 200 of an example embodiment can be implemented.These components are described in more detail below. Ecosystem 101includes a variety of systems and components that can generate and/ordeliver one or more sources of information/data and related services tothe in-vehicle control system 150 and the image processing module 200,which can be installed in the vehicle 105. For example, a camerainstalled in the vehicle 105, as one of the devices of vehiclesubsystems 140, can generate image and timing data that can be receivedby the in-vehicle control system 150. The in-vehicle control system 150and the image processing module 200 executing therein can receive thisimage and timing data input. As described in more detail below, theimage processing module 200 can process the image input and extractobject features, which can be used by an autonomous vehicle controlsubsystem, as another one of the subsystems of vehicle subsystems 140.The autonomous vehicle control subsystem, for example, can use thereal-time extracted object features to safely and efficiently navigateand control the vehicle 105 through a real world driving environmentwhile avoiding obstacles and safely controlling the vehicle.

In an example embodiment as described herein, the in-vehicle controlsystem 150 can be in data communication with a plurality of vehiclesubsystems 140, all of which can reside in a user's vehicle 105. Avehicle subsystem interface 141 is provided to facilitate datacommunication between the in-vehicle control system 150 and theplurality of vehicle subsystems 140. The in-vehicle control system 150can include a data processor 171 configured to execute the imageprocessing module 200 for processing image data received from one ormore of the vehicle subsystems 140. The data processor 171 can becombined with a data storage device 172 as part of a computing system170 in the in-vehicle control system 150. The data storage device 172can be used to store data, processing parameters, and data processinginstructions. A processing module interface 165 can be provided tofacilitate data communications between the data processor 171 and theimage processing module 200. In various example embodiments, a pluralityof processing modules, configured similarly to image processing module200, can be provided for execution by data processor 171. As shown bythe dashed lines in FIG. 1, the image processing module 200 can beintegrated into the in-vehicle control system 150, optionally downloadedto the in-vehicle control system 150, or deployed separately from thein-vehicle control system 150.

Although not illustrated in FIG. 1, the in-vehicle control system 150and/or the vehicle subsystems 140 may include a vehicle operationalsubsystem 300 (e.g., as shown in FIGS. 2 and 3) configured detect anangle between two adjacent sections of an articulated vehicle. In thecase of a semi-truck having a single trailer, the subsystem 300 may beconfigured to detect and determine the angle formed between a tractor ofthe semi-truck and the trailer. Further details regarding the subsystem300 are provided below.

The in-vehicle control system 150 can be configured to receive ortransmit data to/from a wide-area network 120 and network resources 122connected thereto. An in-vehicle web-enabled device 130 and/or a usermobile device 132 can be used to communicate via network 120. Aweb-enabled device interface 131 can be used by the in-vehicle controlsystem 150 to facilitate data communication between the in-vehiclecontrol system 150 and the network 120 via the in-vehicle web-enableddevice 130. Similarly, a user mobile device interface 133 can be used bythe in-vehicle control system 150 to facilitate data communicationbetween the in-vehicle control system 150 and the network 120 via theuser mobile device 132. In this manner, the in-vehicle control system150 can obtain real-time access to network resources 122 via network120. The network resources 122 can be used to obtain processing modulesfor execution by data processor 171, data content to train internalneural networks, system parameters, or other data.

The ecosystem 101 can include a wide area data network 120. The network120 represents one or more conventional wide area data networks, such asthe Internet, a cellular telephone network, satellite network, pagernetwork, a wireless broadcast network, gaming network, WiFi network,peer-to-peer network, Voice over IP (VoIP) network, etc. One or more ofthese networks 120 can be used to connect a user or client system withnetwork resources 122, such as websites, servers, central control sites,or the like. The network resources 122 can generate and/or distributedata, which can be received in vehicle 105 via in-vehicle web-enableddevices 130 or user mobile devices 132. The network resources 122 canalso host network cloud services, which can support the functionalityused to compute or assist in processing image input or image inputanalysis. Antennas can serve to connect the in-vehicle control system150 and the image processing module 200 with the data network 120 viacellular, satellite, radio, or other conventional signal receptionmechanisms. Such cellular data networks are currently available (e.g.,Verizon™, AT&T™, T-Mobile™, etc.). Such satellite-based data or contentnetworks are also currently available (e.g., SiriusXM™, HughesNet™,etc.). The broadcast networks, such as AM/FM radio networks, pagernetworks, UHF networks, gaming networks, WiFi networks, peer-to-peernetworks, Voice over IP (VoIP) networks, and the like are alsoavailable. Thus, the in-vehicle control system 150 and the imageprocessing module 200 can receive web-based data or content via anin-vehicle web-enabled device interface 131, which can be used toconnect with the in-vehicle web-enabled device receiver 130 and network120. In this manner, the in-vehicle control system 150 and the imageprocessing module 200 can support a variety of network-connectablein-vehicle devices and systems from within a vehicle 105.

As shown in FIG. 1, the in-vehicle control system 150 and the imageprocessing module 200 can also receive data, image processing controlparameters, and training content from user mobile devices 132, which canbe located inside or proximately to the vehicle 105. The user mobiledevices 132 can represent standard mobile devices, such as cellularphones, smartphones, personal digital assistants (PDA's), MP3 players,tablet computing devices (e.g., iPad™), laptop computers, CD players,and other mobile devices, which can produce, receive, and/or deliverdata, image processing control parameters, and content for thein-vehicle control system 150 and the image processing module 200. Asshown in FIG. 1, the mobile devices 132 can also be in datacommunication with the network cloud 120. The mobile devices 132 cansource data and content from internal memory components of the mobiledevices 132 themselves or from network resources 122 via network 120.Additionally, mobile devices 132 can themselves include a GPS datareceiver, accelerometers, WiFi triangulation, or other geo-locationsensors or components in the mobile device, which can be used todetermine the real-time geo-location of the user (via the mobile device)at any moment in time. In any case, the in-vehicle control system 150and the image processing module 200 can receive data from the mobiledevices 132 as shown in FIG. 1.

Referring still to FIG. 1, the example embodiment of ecosystem 101 caninclude vehicle operational subsystems 140. For embodiments that areimplemented in a vehicle 105, many standard vehicles include operationalsubsystems, such as electronic control units (ECUs), supportingmonitoring/control subsystems for the engine, brakes, transmission,electrical system, emissions system, interior environment, and the like.For example, data signals communicated from the vehicle operationalsubsystems 140 (e.g., ECUs of the vehicle 105) to the in-vehicle controlsystem 150 via vehicle subsystem interface 141 may include informationabout the state of one or more of the components or subsystems of thevehicle 105. In particular, the data signals, which can be communicatedfrom the vehicle operational subsystems 140 to a Controller Area Network(CAN) bus of the vehicle 105, can be received and processed by thein-vehicle control system 150 via vehicle subsystem interface 141.Embodiments of the systems and methods described herein can be used withsubstantially any mechanized system that uses a CAN bus or similar datacommunications bus as defined herein, including, but not limited to,industrial equipment, boats, trucks, machinery, or automobiles; thus,the term “vehicle” as used herein can include any such mechanizedsystems. Embodiments of the systems and methods described herein canalso be used with any systems employing some form of network datacommunications; however, such network communications are not required.

Referring still to FIG. 1, the example embodiment of ecosystem 101, andthe vehicle operational subsystems 140 therein, can include a variety ofvehicle subsystems in support of the operation of vehicle 105. Ingeneral, the vehicle 105 may take the form of a car, truck, motorcycle,bus, boat, airplane, helicopter, lawn mower, earth mover, snowmobile,aircraft, recreational vehicle, amusement park vehicle, farm equipment,construction equipment, tram, golf cart, train, and trolley, forexample. Other vehicles are possible as well. The vehicle 105 may beconfigured to operate fully or partially in an autonomous mode. Forexample, the vehicle 105 may control itself while in the autonomousmode, and may be operable to determine a current state of the vehicleand its environment, determine a predicted behavior of at least oneother vehicle in the environment, determine a confidence level that maycorrespond to a likelihood of the at least one other vehicle to performthe predicted behavior, and control the vehicle 105 based on thedetermined information. While in autonomous mode, the vehicle 105 may beconfigured to operate without human interaction.

The vehicle 105 may include various vehicle subsystems such as a vehicledrive subsystem 142, vehicle sensor subsystem 144, vehicle controlsubsystem 146, and occupant interface subsystem 148. As described above,the vehicle 105 may also include the in-vehicle control system 150, thecomputing system 170, and the image processing module 200. The vehicle105 may include more or fewer subsystems and each subsystem couldinclude multiple elements. Further, each of the subsystems and elementsof vehicle 105 could be interconnected. Thus, one or more of thedescribed functions of the vehicle 105 may be divided up into additionalfunctional or physical components or combined into fewer functional orphysical components. In some further examples, additional functional andphysical components may be added to the examples illustrated by FIG. 1.

The vehicle drive subsystem 142 may include components operable toprovide powered motion for the vehicle 105. In an example embodiment,the vehicle drive subsystem 142 may include an engine or motor,wheels/tires, a transmission, an electrical subsystem, and a powersource. The engine or motor may be any combination of an internalcombustion engine, an electric motor, steam engine, fuel cell engine,propane engine, or other types of engines or motors. In some exampleembodiments, the engine may be configured to convert a power source intomechanical energy. In some example embodiments, the vehicle drivesubsystem 142 may include multiple types of engines or motors. Forinstance, a gas-electric hybrid car could include a gasoline engine andan electric motor. Other examples are possible.

The wheels of the vehicle 105 may be standard tires. The wheels of thevehicle 105 may be configured in various formats, including a unicycle,bicycle, tricycle, or a four-wheel format, such as on a car or a truck,for example. Other wheel geometries are possible, such as thoseincluding six or more wheels. Any combination of the wheels of vehicle105 may be operable to rotate differentially with respect to otherwheels. The term wheel may generally refer to a structure comprising arim configured to be fixedly attached to a tire, which is typicallyformed of rubber. Optionally, a wheel may include a hubcap attached toan outer surface of the rim or the tire may be exposed to theenvironment without the inclusion of a hubcap. As used herein, thedetection and/or segmentation of a wheel within an image may include thedetection of the entire wheel/tire combination, including the rubbertire and the central wheel, when visible.

The wheels of a given vehicle may represent at least one wheel that isfixedly coupled to the transmission and at least one tire coupled to arim of the wheel that could make contact with the driving surface. Thewheels may include a combination of metal and rubber, or anothercombination of materials. The transmission may include elements that areoperable to transmit mechanical power from the engine to the wheels. Forthis purpose, the transmission could include a gearbox, a clutch, adifferential, and drive shafts. The transmission may include otherelements as well. The drive shafts may include one or more axles thatcould be coupled to one or more wheels. The electrical system mayinclude elements that are operable to transfer and control electricalsignals in the vehicle 105. These electrical signals can be used toactivate lights, servos, electrical motors, and other electricallydriven or controlled devices of the vehicle 105. The power source mayrepresent a source of energy that may, in full or in part, power theengine or motor. That is, the engine or motor could be configured toconvert the power source into mechanical energy. Examples of powersources include gasoline, diesel, other petroleum-based fuels, propane,other compressed gas-based fuels, ethanol, fuel cell, solar panels,batteries, and other sources of electrical power. The power source couldadditionally or alternatively include any combination of fuel tanks,batteries, capacitors, or flywheels. The power source may also provideenergy for other subsystems of the vehicle 105.

The vehicle sensor subsystem 144 may include a number of sensorsconfigured to sense information about an environment or condition of thevehicle 105. For example, the vehicle sensor subsystem 144 may includean inertial measurement unit (IMU), a Global Positioning System (GPS)transceiver, a RADAR unit, a laser range finder/LIDAR unit, and one ormore cameras or image capture devices (e.g., an optical sensor 210 asshown in FIG. 2). The optical sensor may be embodied as a LiDAR detectoror a camera (e.g., a conventional visible wavelength camera). Thevehicle sensor subsystem 144 may also include sensors configured tomonitor internal systems of the vehicle 105 (e.g., an 02 monitor, a fuelgauge, an engine oil temperature). Other sensors are possible as well.One or more of the sensors included in the vehicle sensor subsystem 144may be configured to be actuated separately or collectively in order tomodify a position, an orientation, or both, of the one or more sensors.

The IMU may include any combination of sensors (e.g., accelerometers andgyroscopes) configured to sense position and orientation changes of thevehicle 105 based on inertial acceleration. The GPS transceiver may beany sensor configured to estimate a geographic location of the vehicle105. For this purpose, the GPS transceiver may include areceiver/transmitter operable to provide information regarding theposition of the vehicle 105 with respect to the Earth. The RADAR unitmay represent a system that utilizes radio signals to sense objectswithin the local environment of the vehicle 105. In some embodiments, inaddition to sensing the objects, the RADAR unit may additionally beconfigured to sense the speed and the heading of the objects proximateto the vehicle 105. The laser range finder or LIDAR unit may be anysensor configured to sense objects in the environment in which thevehicle 105 is located using lasers. In an example embodiment, the laserrange finder/LIDAR unit may include one or more laser sources, a laserscanner, and one or more detectors, among other system components. Thelaser range finder/LIDAR unit can be configured to operate in a coherent(e.g., using heterodyne detection) or an incoherent detection mode. Thecameras may include one or more devices configured to capture aplurality of images of the environment of the vehicle 105. The camerasmay be still image cameras or motion video cameras.

The vehicle control system 146 may be configured to control operation ofthe vehicle 105 and its components. Accordingly, the vehicle controlsystem 146 may include various elements such as a steering unit, athrottle, a brake unit, a navigation unit, and an autonomous controlunit.

The steering unit may represent any combination of mechanisms that maybe operable to adjust the heading of vehicle 105. The throttle may beconfigured to control, for instance, the operating speed of the engineand, in turn, control the speed of the vehicle 105. The brake unit caninclude any combination of mechanisms configured to decelerate thevehicle 105. The brake unit can use friction to slow the wheels in astandard manner. In other embodiments, the brake unit may convert thekinetic energy of the wheels to electric current. The brake unit maytake other forms as well. The navigation unit may be any systemconfigured to determine a driving path or route for the vehicle 105. Thenavigation unit may additionally be configured to update the drivingpath dynamically while the vehicle 105 is in operation. In someembodiments, the navigation unit may be configured to incorporate datafrom the image processing module 200, the GPS transceiver, and one ormore predetermined maps so as to determine the driving path for thevehicle 105. The autonomous control unit may represent a control systemconfigured to identify, evaluate, and avoid or otherwise negotiatepotential obstacles in the environment of the vehicle 105. In general,the autonomous control unit may be configured to control the vehicle 105for operation without a driver or to provide driver assistance incontrolling the vehicle 105. In some embodiments, the autonomous controlunit may be configured to incorporate data from the image processingmodule 200, the GPS transceiver, the RADAR, the LIDAR, the cameras, andother vehicle subsystems to determine the driving path or trajectory forthe vehicle 105. The vehicle control system 146 may additionally oralternatively include components other than those shown and described.

Occupant interface subsystems 148 may be configured to allow interactionbetween the vehicle 105 and external sensors, other vehicles, othercomputer systems, and/or an occupant or user of vehicle 105. Forexample, the occupant interface subsystems 148 may include standardvisual display devices (e.g., plasma displays, liquid crystal displays(LCDs), touchscreen displays, heads-up displays, or the like), speakersor other audio output devices, microphones or other audio input devices,navigation interfaces, and interfaces for controlling the internalenvironment (e.g., temperature, fan, etc.) of the vehicle 105.

In an example embodiment, the occupant interface subsystems 148 mayprovide, for instance, capabilities for a user/occupant of the vehicle105 to interact with the other vehicle subsystems. The visual displaydevices may provide information to a user of the vehicle 105. The userinterface devices can also be operable to accept input from the user viaa touchscreen. The touchscreen may be configured to sense at least oneof a position and a movement of a user's finger via capacitive sensing,resistance sensing, or a surface acoustic wave process, among otherpossibilities. The touchscreen may be capable of sensing finger movementin a direction parallel or planar to the touchscreen surface, in adirection normal to the touchscreen surface, or both, and may also becapable of sensing a level of pressure applied to the touchscreensurface. The touchscreen may be formed of one or more translucent ortransparent insulating layers and one or more translucent or transparentconducting layers. The touchscreen may take other forms as well.

In other instances, the occupant interface subsystems 148 may providecapabilities for the vehicle 105 to communicate with devices within itsenvironment. The microphone may be configured to receive audio (e.g., avoice command or other audio input) from a user of the vehicle 105.Similarly, the speakers may be configured to output audio to a user ofthe vehicle 105. In one example embodiment, the occupant interfacesubsystems 148 may be configured to wirelessly communicate with one ormore devices directly or via a communication network. For example, awireless communication system could use 3G cellular communication, suchas CDMA, EVDO, GSM/GPRS, or 4G cellular communication, such as WiMAX orLTE. Alternatively, the wireless communication system may communicatewith a wireless local area network (WLAN), for example, using WIFI®. Insome embodiments, the wireless communication system 146 may communicatedirectly with a device, for example, using an infrared link, BLUETOOTH®,or ZIGBEE®. Other wireless protocols, such as various vehicularcommunication systems, are possible within the context of thedisclosure. For example, the wireless communication system may includeone or more dedicated short range communications (DSRC) devices that mayinclude public or private data communications between vehicles and/orroadside stations.

Many or all of the functions of the vehicle 105 can be controlled by thecomputing system 170. The computing system 170 may include at least onedata processor 171 (which can include at least one microprocessor) thatexecutes processing instructions stored in a non-transitory computerreadable medium, such as the data storage device 172. The computingsystem 170 may also represent a plurality of computing devices that mayserve to control individual components or subsystems of the vehicle 105in a distributed fashion. In some embodiments, the data storage device172 may contain processing instructions (e.g., program logic) executableby the data processor 171 to perform various functions of the vehicle105, including those described herein in connection with the drawings.The data storage device 172 may contain additional instructions as well,including instructions to transmit data to, receive data from, interactwith, or control one or more of the vehicle drive subsystem 142, thevehicle sensor subsystem 144, the vehicle control subsystem 146, and theoccupant interface subsystems 148.

In addition to the processing instructions, the data storage device 172may store data such as image processing parameters, training data,roadway maps, and path information, among other information. Suchinformation may be used by the vehicle 105 and the computing system 170during the operation of the vehicle 105 in the autonomous,semi-autonomous, and/or manual modes.

The vehicle 105 may include a user interface for providing informationto or receiving input from a user or occupant of the vehicle 105. Theuser interface may control or enable control of the content and thelayout of interactive images that may be displayed on a display device.Further, the user interface may include one or more input/output deviceswithin the set of occupant interface subsystems 148, such as the displaydevice, the speakers, the microphones, or a wireless communicationsystem.

The computing system 170 may control the function of the vehicle 105based on inputs received from various vehicle subsystems (e.g., thevehicle drive subsystem 142, the vehicle sensor subsystem 144, and thevehicle control subsystem 146), as well as from the occupant interfacesubsystem 148. For example, the computing system 170 may use input fromthe vehicle control system 146 in order to control the steering unit toavoid an obstacle detected by the vehicle sensor subsystem 144 and theimage processing module 200, move in a controlled manner, or follow apath or trajectory based on output generated by the image processingmodule 200. In an example embodiment, the computing system 170 can beoperable to provide control over many aspects of the vehicle 105 and itssubsystems.

Although FIG. 1 shows various components of vehicle 105, e.g., vehiclesubsystems 140, computing system 170, data storage device 172, and imageprocessing module 200, as being integrated into the vehicle 105, one ormore of these components could be mounted or associated separately fromthe vehicle 105. For example, data storage device 172 could, in part orin full, exist separate from the vehicle 105. Thus, the vehicle 105could be provided in the form of device elements that may be locatedseparately or together. The device elements that make up vehicle 105could be communicatively coupled together in a wired or wirelessfashion.

Additionally, other data and/or content (denoted herein as ancillarydata) can be obtained from local and/or remote sources by the in-vehiclecontrol system 150 as described above. The ancillary data can be used toaugment, modify, or train the operation of the image processing module200 based on a variety of factors including, the context in which theuser is operating the vehicle (e.g., the location of the vehicle, thespecified destination, direction of travel, speed, the time of day, thestatus of the vehicle, etc.), and a variety of other data obtainablefrom the variety of sources, local and remote, as described herein.

In a particular embodiment, the in-vehicle control system 150 and theimage processing module 200 can be implemented as in-vehicle componentsof vehicle 105. In various example embodiments, the in-vehicle controlsystem 150 and the image processing module 200 in data communicationtherewith can be implemented as integrated components or as separatecomponents. For example, the image processing module 200 can be includedas a set of instructions stored in a non-transitory computer readablemedium, such as the data storage device 172, for causing the dataprocessor 171 to perform various image processing functionality. In anexample embodiment, the software components of the in-vehicle controlsystem 150 and/or the image processing module 200 can be dynamicallyupgraded, modified, and/or augmented by use of the data connection withthe mobile devices 132 and/or the network resources 122 via network 120.The in-vehicle control system 150 can periodically query a mobile device132 or a network resource 122 for updates or updates can be pushed tothe in-vehicle control system 150.

Systems and Methods for Determining Tractor to Trailer Angle

In the various example embodiments disclosed herein, a system and methodare provided for detecting the angle between adjacent sections of anarticulated vehicle, which may be used for autonomous driving of thevehicle. Embodiments of the articulated vehicle include a semi-truckhaving a tractor and at least one trailer, an articulated bus, a train,etc.

FIG. 2 is a simplified diagram of a semi-truck having a vehicleoperational subsystem configured to detect an angle between a tractorand a trailer of the semi-truck in accordance with aspects of thisdisclosure. As shown in FIG. 2, the semi-truck 200 includes a tractor205 coupled to a trailer 215. The trailer 215 may include a container220 in which goods are configured to be stored during transportation.The trailer 215 may be connected to the tractor 205 at a pivot point,for example, via a fifth wheel configuration including a fifth wheelcoupling device (not illustrated) installed on the tractor 205 and akingpin (not illustrated) installed on the trailer 215. However, otherembodiments may employ other configurations used to couple the trailer215 to the tractor 205. Further, the tractor 205 may include vehicleoperational subsystems 140 (not illustrated) and/or an in-vehiclecontrol system 150 (not illustrated) including a vehicle operationalsubsystem 300.

Since the trailer 215 is configured to pivot around the coupling pointwith the tractor 205 (e.g., at the fifth wheel connection point), theangle formed between the trailer 215 and the tractor 205 may vary as thetractor 205 moves or is driven autonomously. The current angle betweenthe tractor 205 and the trailer 215 is a significant variable which canbe used to define the current state of the semi-truck 200, since theangle may affect the overall center of gravity of the semi-truck 200,the path taken by the trailer 215 during driving, the wind resistance ofthe semi-truck 200, etc. An accurate measurement of the angle betweenthe tractor 205 and the trailer 215 may also be of significance duringsharp turns, where there is a possibility that the trailer 215 maycollide with a portion of the tractor 205, and during hard breaking ofthe semi-truck 200, which has the potential to lead to jackknifing ofthe trailer 215. Since jackknifing in particular can be dangerous,detection of the conditions under which jackknifing can occur, includingthe tractor 205 to trailer 215 angle can improve the safety of thevehicle when driven autonomously.

One possible technique for measuring the angle between the tractor 205and the trailer 215 may involve installing an angular sensor on thekingpin of the trailer 215 configured to measure the angle between thetrailer 215 and the tractor 205. However, there may be a number ofdrawbacks associated with the angular sensor installed on the trailer215. For example, the fifth wheel coupling between the tractor 205 andthe trailer 215 may involve the transmission of large forces through thefifth wheel coupling mechanism, some of which may be imparted on theangular sensor. These forces may lead to early wear and tear on theangular sensor, which may require frequent replacement due to highfailure rates of the angular sensor. Additionally, the logisticsassociated with transportation using semi-trucks may involve the tractor205 coupling with and hauling a number of different trailers 215, eachof which may be manufactured by a different producer. Thus, the angularsensors installed in each trailer 215 would have to be manuallyconnected and set up to properly interface with the in-vehicle controlsystem 150 in the tractor 205. This set up involves extra time andexpertise for the driver and/or loading staff, leading to a source ofinefficiency when coupling/uncoupling trailers 215 from the tractor 205.

Additionally, the use of an angular sensor installed on the trailer 215may be capable of providing only a single point of measurement of thetractor 205 to trailer 215 angle. Thus, if the measurement accuracy ofthe angular sensor deteriorates, all measurements provided to thein-vehicle control system 150 will suffer, reducing the ability of thein-vehicle control system 150 to autonomously control the semi-truck200.

Aspects of this disclosure may address one or more of theabove-described drawbacks. For example, the tractor 205 may include anoptical sensor 210. In certain embodiments, the optical sensor 210 isinstalled facing the trailer 215. The optical sensor 210 may face thetrailer 215 from the rear of the tractor 215 to obtain image(s) of thetrailer. With continued reference to FIG. 2, an X-axis and a Y-axis maybe defined for each of the optical sensor 210 and a reference point onthe trailer 215. In some embodiments, an angle 230 between the opticalsensor X-axis and the trailer X-axis may be defined as the angle betweenthe tractor 205 and the trailer 215.

Depending on the implementation, the optical sensor 210 may be embodiedas a LiDAR detector 211 or a camera 213 (e.g., a conventional visiblewavelength camera). However, aspects of this disclosure are not limitedto these embodiments and the optical sensor 210 may also be embodied asan infrared camera, a multi spectrum camera, a RADAR, etc.

The optical sensor 210 may provide raw or partially processed data toone or more processors which may determine the angle 230 using thereceived data. The one or more processors may include a processorintegrated in the optical sensor 210, the data processor 171, and/or aprocessor included in the vehicle subsystems 140. For the sake ofclarity, the set of one or more processors may be referred to simply asthe singular “processor” herein.

FIG. 3 illustrates an example block diagram of the vehicle operationalsubsystem for the autonomous control of the semi-truck that has theangle 230 illustrated in FIG. 2 in accordance with aspects of thisdisclosure. The illustrated system 300 may include the optical sensor210. In certain embodiments, the optical sensor 210 is configured tooutput optical data indicative of the angle of the trailer 215 withrespect to the tractor 205. The optical sensor 210 may be installed onthe tractor 205 as shown in FIG. 2, and thus, the optical data mayinclude data representative of at least a portion of the surface of thetrailer 215 visible from the point of view of the optical sensor 210. Insome embodiments, the optical data may include point cloud dataincluding a set of data points representative of the location of asurface of the trailer 215 in space.

At block 310, a processor may determine the vehicle state using theoptical sensor data received from the optical sensor 210. This mayinclude determining the angle (e.g., the angle 230) between the trailer215 and the tractor 205. In addition, the vehicle state determined inblock 310 may include other variables defining a current (or past) stateof the semi-truck 200, including the semi-truck's 200 speed,acceleration/breaking, current lane position, GPS position, etc. Atblock 320, the processor may use the determined vehicle state and adesired trajectory 315 to determine appropriate vehicle control commands320 to autonomously drive the semi-truck 200 in accordance with thedesired trajectory 315.

FIG. 4 illustrates an example method which can be used to determine theangle 230 illustrated in FIG. 2 in accordance with aspects of thisdisclosure. The method 400 begins at block 401. At block 405, theprocessor receives point cloud data from the optical sensor 210illustrated in FIG. 2. However, in other embodiments, the processor mayreceive optical data from the optical sensor 210, which may include datarepresentative of at least a portion of the surface of the trailer 215visible from the point of view of the optical sensor 210 in a form otherthan a point cloud. In some of these embodiments, the processor may beconfigured to generate a point cloud including a set of data pointsrepresentative of the location of a surface of the trailer 215 in spacefrom the optical sensor data.

At block 410, the processor determines side information of the trailer215 based on the point cloud data received from the optical sensor 210.As used herein, side information may refer to data representative of theshape of the external surfaces of the trailer 215. This may involvedetecting markers on the side of the container 220 using an approachwhich is based on the type of the optical sensor 210. For example, whenthe optical sensor 210 is embodied as a LiDAR detector 211, theprocessor may extract points from the point cloud data with areflectance value lower than a threshold reflectance value. Theprocessor may then group the extracted points into clusters based on thedistance between the extracted points. For example, points that arewithin a threshold distance from each other may be grouped together. Theprocessor may also extract shape feature(s) from each cluster of points.The extraction of shape feature(s) may be performed using any techniquefor identifying shapes from clusters of points, for example, usingstandard image processing techniques which identify shape feature(s) inimages. The processor may also identify and/or assign a marker ID foreach of the clusters of points based on the extracted shape feature(s).

In the embodiment where the optical sensor 210 is embodied as the camera213, the processor may perform an alternative technique at block 410.For example, the processor may detect a quick response (QR) code in theimage. In this embodiment, one or more QR codes may be attached (e.g.,via magnets, adhesive, etc.) to the trailer 215 within the expectedfield of view of the camera 213. The camera 213 may identify the QRcodes as a part of determining the side information of the trailer 215.The processor may further determine QR code 3D position data usingprojected LiDAR points from the LiDAR detector 211 to cameracalibration.

At block 415, the processor extracts points from the point cloud datathat belong to the trailer 215. In some embodiments, the processor mayuse a random sample consensus (RANSAC)-based plane fitting algorithm todetect all the candidate 3D planes in the point cloud. That is, theprocessor may determine a set of candidate 3D planes which can be usedto represent the plane of the visible side of the trailer 215 that areconsistent with the point cloud data obtained in block 405. For each ofthe candidate planes, the processor may calculate the distance from theplane to the closest marker determined in block 410. The processor mayalso, for each of the candidate planes, determine that the candidateplane belongs to the container 220 when the distance from the plane tothe closest marker is less than a predefined threshold distance. Theprocessor may extract K container planes (where) P_(k), k∈[1,K]) fromall the candidate planes determined to belong to the container 220. Theprocessor may also construct a container planes point cloud byextracting points from the point cloud data that belong to the extractedcontainer planes. For each points X_(i), the processor may store amapping k=m(X_(i)) for the correspondence between the points X_(i) andthe index(k) of the corresponding container plane.

At block 420, the processor removes points from the point cloud datawhich belong to edge areas of the trailer 215. For example, for eachpoint in the container planes point cloud, the processor may use thecurrent point as a reference point and determine the reference point's Nclosest points. The processor may define the N closest points asneighboring points for the reference point. The processor may also useleast square fitting to fit a 3D plane from the neighboring points ofthe reference point. The processor may also calculate the sum ofdistance from neighboring points to the fitted plane and, when the sumof distance is greater than a predefined threshold, the processor maydetermine that the reference point belongs to the edge areas. Theprocessor may then remove points determined to belong to the edge areasfrom the container planes point cloud.

At block 425, the processor refines 3D trailer planes models. In certainembodiments, an objective function may be used to optimize the 3Dtrailer points. For example, in some implementations, the 3D trailerpoints may comprise noise. However, since the general shape of thetrailer 215 in the real world will not have noise (e.g., the shapes oftrailers 215 are predictable), the processor can construct the objectivefunction to refine the 3D trailer planes models by reducing the noise inthe 3 trailer points that is inconsistent with the expected trailer 215shape. In certain embodiments, the objective function may include one ormore of the following variables: the models of K container planes, N 3Dpoints in container planes point cloud, the mapping stored in block 420,a function that relates the distance from 3D point X_(i) to plane, and apenalty function for violating the constraint that the normals of planeare perpendicular. By optimizing the objective function, the processorcan refine the 3D trailer planes model.

At block 430, the processor determines the angle 230 from the refined 3Dtrailer planes. In some embodiments, the processor can determine theangle 230 from the normals of the optimal 3D container planes determinedin block 425. The method 400 ends at block 435.

FIG. 5 illustrates another example method which can be used to determinethe angle 230 illustrated in FIG. 2 in accordance with aspects of thisdisclosure. The method 500 begins at block 501. At block 505, theprocessor receives point cloud data from the optical sensor 210illustrated in FIG. 2.

At block 510, the processor determines side information of the trailer215 based on the point cloud data received from the optical sensor 210.Block 510 may be implemented in a similar fashion to block 410 of FIG.4. This may involve a different technique depending on the particularembodiment of the optical sensor 210 (e.g., whether embodied as theLiDAR detector 211 or the camera 213).

At block 515, the processor extracts points from the point cloud datathat belong to the trailer 215. In some embodiments, the processor mayuse a random sample consensus (RANSAC)-based plane fitting algorithm todetect all the candidate 3D planes in the point cloud. That is, theprocessor may determine a set of candidate 3D planes which can be usedto represent the plane of the visible side of the trailer 215 that areconsistent with the point cloud data obtained in block 505. For each ofthe candidate planes, the processor may calculate the distance from theplane to the closest marker determined in block 510. The processor mayalso, for each of the candidate planes, determine that the candidateplane belongs to the container 220 when the distance from the plane tothe closest marker is less than a predefined threshold distance. Theprocessor may extract K container planes (where P_(k), k∈[1,K]) from allthe candidate planes determined to belong to the container 220. Theprocessor may also construct a container planes point cloud byextracting points from the point cloud data that belong to the extractedcontainer planes.

At block 520, the processor refines 3D trailer planes models using anexpectation-maximization (EM) framework. The EM framework may includeperforming an iteration of a process. For example, at the E-step of thet iteration, the processor may determine a mapping function m^(t)(X_(i))to determine the index of the corresponding container plane for 3D pointFor each 3D point X_(i), the mapping function m^(t)(X_(i)) may be thedistance from the 3D point X_(i) to the plane P^(t−1) _(j). The planemay be the model of the container planes at the t−1 iteration. At theM-step of the t iteration, an objective function may be defined whichmay be optimized to refine the 3D trailer points. After iteratingthrough the steps of the objective function, the objective function mayconverge on a stable result, thereby optimizing the 3D trailer points.In certain embodiments, the objective function may include one or moreof the following variables: the models of K container planes at the t−1iteration, N 3D points in container planes point cloud, the mappingdetermined during the E-step of the iteration, the function that relatesthe distance from 3D point X_(i) to plane, and a penalty function forviolating the constraint that the normals of plane are perpendicular. Byoptimizing the objective function, the processor can refine the 3Dtrailer planes model. The E-step and M-step may be iterated untilconvergence to obtain the final optimal 3D container planes models.

At block 525, the processor determines the angle 230 from the refined 3Dtrailer planes. In some embodiments, the processor can determine theangle 230 from the normals of the optimal 3D container planes determinedin block 520. The method 500 ends at block 535.

While there have been shown and described and pointed out thefundamental novel features of the invention as applied to certaininventive embodiments, it will be understood that the foregoing isconsidered as illustrative only of the principles of the invention andnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Modifications or variations are possible in light ofthe above teachings. The embodiments discussed were chosen and describedto provide the best illustration of the principles of the invention andits practical application to enable one of ordinary skill in the art toutilize the invention in various embodiments and with variousmodifications as are suited to the particular use contemplate. All suchmodifications and variations are within the scope of the invention asdetermined by the appended claims when interpreted in accordance withthe breadth to which they are entitled.

What is claimed is:
 1. An in-vehicle control system for a vehicleincluding a tractor and trailer, comprising: an optical sensorconfigured to be mounted on the tractor and generate optical dataindicative of an angle formed between the trailer and the tractor; aprocessor; and a computer-readable memory in communication with theprocessor and having stored thereon computer-executable instructions tocause the processor to: receive the optical data, extract datarepresentative of one or more markers on a surface of the trailer fromthe optical data, determine at least one candidate plane representativeof the surface of the trailer based on the extracted data, and determinethe angle between the trailer and the tractor based at least in part onthe at least one candidate plane and the one or more markers.
 2. Thein-vehicle control system of claim 1, wherein the memory further hasstored thereon computer-executable instructions to cause the processorto: identify data points within the optical data having a reflectancevalue lower than a threshold reflectance value, wherein the extractionof the data representative of one or more markers on a surface of thetrailer comprises extracting the identified data points.
 3. Thein-vehicle control system of claim 1, wherein a field of view of theoptical sensor includes at least a portion of the surface of thetrailer.
 4. The in-vehicle control system of claim 1, wherein theoptical sensor comprises a LiDAR detector.
 5. The in-vehicle controlsystem of claim 1, wherein the memory further has stored thereoncomputer-executable instructions to cause the processor to: determine atleast one vehicle control command for autonomously driving the vehiclebased at least in part on the angle between the trailer and the tractor.6. The in-vehicle control system of claim 5, wherein the memory furtherhas stored thereon computer-executable instructions to cause theprocessor to: receive a desired trajectory of the vehicle, wherein thedetermination of the at least one vehicle control command forautonomously driving the vehicle is further based at least in part onthe desired trajectory of the vehicle.
 7. The in-vehicle control systemof claim 1, wherein the optical data comprises point cloud datarepresentative of a location of the surface of the trailer in space. 8.The in-vehicle control system of claim 1, wherein the memory further hasstored thereon computer-executable instructions to cause the processorto: remove data associated with edge areas of the trailer from theoptical data, wherein the extracted data used to determine the at leastone candidate plane does not include the removed data.
 9. The in-vehiclecontrol system of claim 1, wherein the memory further has stored thereoncomputer-executable instructions to cause the processor to: group theextracted data representative of the one or more markers into aplurality of clusters.
 10. The in-vehicle control system of claim 9,wherein the memory further has stored thereon computer-executableinstructions to cause the processor to: extract one or more shapefeatures from each cluster of the plurality of clusters, and assign amarker ID for each cluster based on the one or more shape features. 11.The in-vehicle control system of claim 10, wherein the extraction of theone or more shape features is performed using an image processingtechnique for identifying the one or more shape features in an image.12. A non-transitory computer readable storage medium having storedthereon instructions that, when executed, cause at least one computingdevice to: receive optical data from an optical sensor, the opticalsensor configured to be mounted on a tractor and generate the opticaldata indicative of an angle formed between a trailer and the tractor;extract data representative of one or more markers on a surface of thetrailer from the optical data, determine at least one candidate planerepresentative of the surface of the trailer based on the extracteddata, and determine the angle between the trailer and the tractor basedat least in part on the at least one candidate plane and the one or moremarkers.
 13. The non-transitory computer readable storage medium ofclaim 12, wherein the one or more markers comprise a quick response (QR)code.
 14. The non-transitory computer readable storage medium of claim13, wherein the QR codes are attached to the surface of the trailer viamagnets or an adhesive.
 15. The non-transitory computer readable storagemedium of claim 13, further having stored thereon instructions that,when executed, cause at least one computing device to: determine 3Dposition data for each of the QR codes using projected LiDAR points froma LiDAR detector.
 16. The non-transitory computer readable storagemedium of claim 12, wherein the determination of the at least onecandidate plane comprises using a random sample consensus (RANSAC)-basedplane fitting algorithm to detect the at least one candidate plane inthe extracted data.
 17. The non-transitory computer readable storagemedium of claim 12, further having stored thereon instructions that,when executed, cause at least one computing device to: calculate adistance from each of the at least one candidate plane to a closest oneof the one or more markers.
 18. The non-transitory computer readablestorage medium of claim 17, further having stored thereon instructionsthat, when executed, cause at least one computing device to: determinethat the at least one candidate plane belongs to the trailer in responseto the calculated distance being less than a predefined thresholddistance.
 19. A method for determining the distance between a vehicleand a lane, comprising: receiving optical data from an optical sensor,the optical sensor configured to be mounted on a tractor and generatethe optical data indicative of an angle formed between a trailer and thetractor; extracting data representative of one or more markers on asurface of the trailer from the optical data, determining at least onecandidate plane representative of the surface of the trailer based onthe extracted data, and determining the angle between the trailer andthe tractor based at least in part on the at least one candidate planeand the one or more markers.
 20. The method of claim 19, furthercomprising: refining the at least one candidate plane using anexpectation-maximization framework comprising a plurality of iterationsof: i) an expectation step comprising determining a mapping function andii) a maximization step comprising optimizing an objective function.